Do Faster Routers Imply Faster Communication?

Despite signiicant improvements in network interfaces and software messaging layers, software communication overhead still dominates the hardware routing cost in most parallel systems. In this study, we identify the sources of this overhead by relating user communication services to particular network hardware features. Based on a detailed analysis of the active messages layer on the CM-5, we assign the software messaging cost to speciic user communication services and network features. Our study shows that 50{70% of the software cost of messaging can be attributed to providing end-to-end ow control, in-order delivery, and reliable transmission services. This overhead is a direct eeect of speciic network features { arbitrary delivery order, nite buuering, and limited fault-handling { and is unlikely to be eliminated through improved software implementations. We conclude that reducing this software overhead requires changing the constraints on messaging layers: we propose designing routing networks which replace (or make unnecessary) software to implement user services. Speciically, networks which provide ordering , end-to-end ow control, and hardware-supported fault-tolerance can signiicantly reduce the end-to-end cost of communication.